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SIMILAR STUDENTS, DIFFERENT RESULTS: 

W\^ Do Some. Schools Do Better ? 



The summary report from a large-scale survey of California elementary schools serving low-income students 

Some people say that you can predict the performance of a California school based on its zip code. It is true 
that parent education and socioeconomic level — and a student’s proficiency in English — are important to 
students’ academic success. But school and district practices and policies contribute as well. 

Among schools that serve roughly the same kinds of students in California, a large and consistent gap in 
academic performance exists. This gap can be as much as 250 points on the 200-to-1000 scale of the 
Academic Performance Index (API), the state’s primary accountability measure. That observation prompted 
a two-year collaborative research project overseen by EdSource. The study looked at factors that might 
explain that gap in API scores, focusing on a subset of California elementary schools serving largely low- 
income students. The goal was to determine which current K-5 practices and policies are most strongly 
associated with the higher levels of student performance some schools achieve. 



The research question: What accounts for the difference in API performance among schools that serve similar students? 

As this chart shows. Academic Performance Index (API) scores for California elementary schools facing similar challenges (as indicated 
by the School Characteristics Index or SCI) regularly vary by as much as 250 points. The central research question for this study was: 
What school factors might explain this variance? 



The study looked at a relatively 
narrow band of elementary 
schools to control for student 
characteristics. We chose 
schools that fell between the 
25th and 35th percentile on 
the SCI. As a group, those schools 
tended to have higher-than- 
average percentages of students 
who are English learners, who 
are from low-income families, 
and whose parents are not 
high school graduates. 
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EdSource^ is a not-for-profit 501(c)(3) organization estabiished in California in 1977. 

Independent and impartial, EdSource strives to advance the common good by developing and widely distributing trustworthy, useful 
information that clarifies complex K-12 education issues and promotes thoughtful decisions about California’s public school system. 
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About this Summary Report 

This EdSource report summarizes the Initial Findings issued in October 2005 from a two-year study 
conducted by EdSource, Stanford University, University of Caiifornia-Berkeiey, and American institutes for 
Research (AiR). in addition, this report inciudes information coiiected during the study but not previousiy 
reported (from superintendent interviews and principai open-ended questions) asweii as the resuits of an 
additionai anaiysis conducted on curricuium program choice and association with Academic Performance 
index (APi) scores. Whiie the officiai study findings were the work of the entire team, EdSource takes fuii 
responsibiiityfor the contents of this summary and for any errors or misinterpretations it may contain. 
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In 2005 principals from 257 California 
elementary schools — and more than 5,500 
teachers from those schools — completed 
surveys that asked about a wide range of 
classroom, school, and district practices. The 
survey included 350 — 400 items that were 
grouped into broader domains that reflected 
existing research about effective schools and 
districts. Each represented a group of 
specific behaviors thought to affect student 
performance. The schools in turn were 
grouped into high-, medium-, and low- 
scoring categories based on their API. Their 
answers were compared using a statistical 
method designed to isolate the effect of the 
various domains of teacher, principal, and 
school district practice. 

The study found that four specific 
domains, or clusters of practices, were most 
strongly correlated with higher school API 
scores: I) prioritizing student achievement; 2) 
implementing a coherent, standards-based 
instructional program; 3) using assessment 
data to improve student achievement and 
instruction; and 4) ensuring the availability of 
instructional resources. The remaining three 
domains examined — involving and support- 
ing parents, encouraging teacher collaboration 
and professional development, and enforcing 
high expectations for student behavior — had 
much weaker but still positive correlations 
with school performance. 

The four “effective schools” domains did 
not operate independently but tended to 
occur together in the same school. Therefore, 
a central message of the study is that no single 
action, or even category of actions, can alone 
provide a clear advantage related to student 
performance. Rather, schools that have, on 
average, higher API scores also report more 
strongly that they implement these multiple, 
related practices. 

The study also strongly suggests that 
performance is higher in schools in which the 
actions of teachers, principals, and school 
district officials are all closely aligned and 
tightly focused on student achievement. 
Responses from school principals to an open- 
ended survey question and interviews with 20 
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About the Schools and Districts in this Study 



There were 550 schools with a School Characteristics Index (SCI) that fell between the 25th and 
35th percentiles. The schools in the study were a representative sample from this band, with both 
school and student characteristics roughly similar to those not included in the study. 

The 257 schools that participated in this study came from 145 different school districts, and 56 
of those districts had more than one school participate. 

About a third of the districts and more than 40% of the schools were located in the Los Angeles metro- 
politan area (including San Bernardino and Riverside), with the remainder of schools and districts 
distributed across the rest of Southern California, the Central Valley, and Northern California. 

A third of the schools were in elementary districts, 15% operated on year-round calendars, and 98% 
received Title I funding. 

In the sample, 31% of students had parents who were not high school graduates, compared to 21% 
of elementary students in the state as a whole. 

Comparisons of other student characteristics associated with student achievement are similar. 
Statewide, California’s elementary school population in 2004-05 included 32% English learners, 
50% Hispanics/Latinos, 8% African Americans, 29% whites, 8% Asians, and 57% low-income 
students (based on participation in the free and reduced-priced meal program).The schools in this 
study had higher proportions of English learners, Hispanics, and low-income students. Their total 
student population included 42% English learners, 66% Hispanics, 8% African Americans, 15% 
whites, 6% Asians, and 78% low-income students. These percentages varied from school to school. 

The 2005 Growth API for the average school participating in the study was 702, with a range from 
569 to 821. 

On the statewide rankings from Ito 10 for 2004 Base API, none of the sample schools ranked 10, 
and just 4% ranked 7 or higher. Conversely, 7% were ranked in the bottom decile. The remaining 
89% of schools ranked from 2-6, with the majority a 3 or 4. 



superintendents helped to illuminate 
this dynamic. 

The Similar Students, Different Results 
study was a collaborative effort of 
EdSource, Stanford University, Univer- 
sity of California— Berkeley, and the 
American Institutes for Research 
(AIR). Initial findings were released in 
October 2005. Comprehensive materi- 
als, including those and subsequent 
findings released in April 2006, are 
available online at: www.edsource.org/ 
pub_abs_simstu05.cfin 

This summary report describes the 
work of the entire team and draws 
heavily on the collective work of other 



team members. However, EdSource 
takes full responsibility for its final 
form and any inaccuracies or misrepre- 
sentations it might contain. Along with 
a brief description of the research proj- 
ect itself, this summary provides an 
overview of the initial research findings 
as well as material released since, 
placing them within the context of 
California’s standards-based reform 
efforts. It also looks at how state and 
federal policies related to student 
performance appear to be playing out 
in elementary schools serving low- 
income students and explores some 
possible implications for state policy- 



makers, local school district leaders, 
and educators at elementary schools 
throughout California. 

What we hoped to learn 

From its inception, this study was 
designed to look at which “effective 
schools’’ practices previously identi- 
fied in literature might make the most 
difference in school performance 
under standards-based reform in Cali- 
fornia. That required examining a 
wide breadth of topics across multiple 
levels of the education system, includ- 
ing classrooms, schools, and districts. 
Given such an ambitious approach to 
the topics to be studied, we decided to 
focus our analysis at the school level. 
We also limited the study to elemen- 
tary schools, and further, to those that 
served largely low-income students 
and faced similar levels of challenge. 

The state’s School Characteristics 
Index (SCI) proved useful for identi- 
fying similar schools in terms of the 
students that they served. The band of 
about 550 schools in the 25th — 35th 
percentile range on the SCI in 
2003—04 thus became the subject of 
this study. Most of the schools in that 
group serve high numbers of low- 
income and English learner students, 
and many have high percentages of 
Hispanic/Latino and/or African 
American populations. (See the box 
on this page that explains the charac- 
teristics of the schools in the study.) 

Almost half of the schools in the 
band participated in the study, with 
the principal and at least 80% of the 
K— 5 classroom teachers at the vast 
majority of schools returning surveys. 
In total, 257 principals and more 
than 5,500 teachers participated by 
responding to 350—400 survey items 
on a multitude of school and district 
practices. 

The survey content was first based 
on a review of the research literature 
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Aspects of California’s accountability system relate to this study 



California Standards Tests (CSTs) and CAT/6 

California’s state assessment system annually tests all students in grades 2-11. At the center of the 
system are the California Standards Tests (CSTs). At the elementary level, the CSTs are primarily based 
on the state’s academic content standards in English language arts and math. The state also adminis- 
ters a nationally normed achievement test, the California Achievement Tests, Sixth Edition Survey 
(CAT/6) at grades 3 and 7. 

The Academic Performance Index (API) 

The state uses the CSTs to determine whether schools and districts have met targets for adequate yearly 
progress (AYP) as required by the No Child Left Behind Act (NCLB). In addition, they are used to calcu- 
late each school’s API (with, to a much lesser degree, CAT/6 test results). 

The API is the cornerstone of California’s school accountability system. With rare exceptions, every 
school in California is assigned an API score between 200 and 1000 and also receives a score for each 
“numerically significant’’ subgroup of students, categorized by ethnicity and socioeconomic factors 
(such as poverty). 

A school’s API score is used to rank it statewide among all schools of the same type (elementary, 
middle, and high) and to compare it to the 100 schools most like it in terms of student background 
and other relevant factors. The state uses the School Characteristics Index (described below) for this 
purpose. The state also gives each school a growth target for improving its API. 

Eor this study, the research team chose the 2005 Grovrth API score as the dependent variable for meas- 
uring school performance. The results and findings were also analyzed against CST results averaged 
across grade levels in each school and against a weighted calculation of API score growth over three 
years. The results were similar in each case. 

The School Characteristics Index (SCI) 

The SCI enables state officials to directly compare schools that face a similar level of challenge. These 
comparisons result in “similar school rankings” that adjust for student and school characteristics most 
strongly correlated with test score performance. 

At the time of this study, the student characteristics in the SCI included ethnicity, English learner status, 
length of school attendance at the current school (if less than a year), parent education level, and 
participation in free/reduced-priced meal programs. School characteristics in the SCI included 
participation in a multitrack, year-round schedule, average class sizes, and percentages of teachers 
with full and emergency credentials. 

Of all these factors, parent education level is most strongly correlated with student performance. For 
that reason, it is given the greatest weight in the SCI calculation. 



on effective schools and on high- 
performing, high-poverty schools. 
With that as the backdrop, the goal 
was to explore how the nature of 
elementary school practices in Califor- 



nia was changing in response to 
standards-based reforms, particularly 
state and federal accountability poli- 
cies. Further, we wanted to see how 
much those practices differed between 



schools with high and low perform- 
ance based on the state’s API and to 
what extent various classroom, school, 
and district practices might correlate 
with performance. 

We also were committed to asking 
about “actionable items’’ — clear and 
specific practices that other local 
educators could implement. The 
survey questions asked teachers about 
their practices in the classroom and 
schoolwide, and about the role of 
their principal. Principals reported on 
their management practices and prior- 
ities, on the effectiveness of the 
teachers in their schools, and on the 
policies, practices, and expectations of 
their school districts. 

The survey questions covered a 
wide range of topic areas including 
school context, role of the principal 
and district, core instruction, instruc- 
tional strategies, assessment and data, 
professional development, and the 
respondents’ professional backgrounds. 
The surveys also asked specific ques- 
tions about kindergarten and English 
Language Development (ELD) 
programs and instruction. (The actual 
survey instruments are posted on the 
EdSource website: www.edsource.org/ 
pub_abs_sLmstu05.cfm) 

While school-level performance, 
as measured by the API, was the focus 
of this study, we were interested in the 
broader context within which schools 
operate. We believed that the inter- 
relationships between classroom, 
school, and district practices could 
help shed light on why some schools 
serving low-income children do so 
much better than others. To that end, 
we asked principals several questions 
about the role of the district. As a 
complement to the principal and 
teacher surveys, the research team 
also interviewed 20 school district 
superintendents from 17 districts 
throughout California. These success- 
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ful district leaders were asked to 
discuss the three most effective strate- 
gies their district employed to help 
improve student achievement at high- 
poverty schools. The box on this page 
describes the results of those inter- 
views generally. More details about 
what superintendents said are scat- 
tered throughout this report. 

Initial findings suggest that a 
combination of strategies is necessary 

Education researchers who have 
looked at the impact of various school 
reforms on student achievement 
consistently come to at least one 
shared conclusion: instructional 

improvement is unlikely to result from 
a single policy or practice. This study’s 
findings echo that important point 
but place it within the context of the 
state’s standards-based reform agenda. 

In the late 1990s, the state of 
California began implementing a set 
of reforms intended to improve 
student achievement. They were built 
on the ideas of high academic con- 
tent standards, the measurement of 
performance using standardized tests, 
and public accountability for schools 
based on student test results. Taken 
together, those reforms appear to be 
affecting schools’ instructional prac- 
tices. For example, nearly 100% of 
the principals and 94% of the teach- 
ers surveyed reported that classroom 
instruction in their schools is guided 
by the state standards. Yet this study 
also makes it clear that the cumulative 
effect of the state’s policies on school 
practices differs among schools. It 
also identifies at least some of the 
attitudes and activities that set higher- 
performing schools apart. And it 
reinforces the conclusion that 
improvement within that standards- 
based environment is correlated with 
the implementation of multiple 
educational practices. (See the box. 




Superintendent interviews suppiemented the schooi surveys 

In addition to surveying principals and teachers, the research team interviewed 20 school district 
administrators regarding their district’s most effective strategies for school improvement. The districts 
selected for interviews all had at least two schools participating in the larger study. Beyond that, they 
also either had at least one high- and one low-performing school, or they had a school with a state API 
rank of seven (the highest rank for this band of low-income schools). 

During face-to-face or telephone interviews, we asked what three strategies the superintendents 
considered most effective in improving student achievement at schools serving high percentages of 
low-income students. They were also asked to describe how those strategies were implemented at the 
school level. In the course of these conversations, the interviewees often mentioned a number of strate- 
gies beyond their top three choices. The following list is based on those most often cited as effective, 
but also notes what proportion of all the superintendents mentioned each strategy: 

1 . Data and assessment: cited as a top strategy by nine respondents and mentioned by 19. 

2 . Professional development: cited as a top strategy by seven superintendents and mentioned by 15. 

3 . Curriculum package: cited as a top strategy by five superintendents and mentioned by 18. 

4 . Role of the principal: cited as a top strategy by three superintendents and mentioned by 18. 

5 . School culture/high expectations: cited as a top strategy by three superintendents and mentioned 
by 15. 

6 . Instruction: cited as a top strategy by three superintendents and mentioned by 13. 

Conducted separately from the survey and analysis, these interviews were meant to provide additional 
understanding of the district context in which our sample schools operated. The information is qualita- 
tive in nature and does not represent a systematic examination of these districts or their practices. 
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Overview of the Research Process 



Methodology 

• Working from the 350-400 survey items asked of teachers and principaison separate surveys, the 
research team grouped sets of reiated practices and poiicies together into seven broad “effective 
schoois” domains. For exampie, questions reiated to use of data were grouped together. Each 
domain was made up of severai smaiier ciusters of reiated items or subdomains, in addition to 
running simpie correiations between aii survey items and schooi APi, the team anaiyzed the reia- 
tionship between APi and each of these subdomains and domains. 

• This study used muitipie regression anaiyses-a statisticai tooi that identifies correiations among vari- 
abies in iarge databases. It holds constant every variable except the ones being examined to show 
how those relate to each other. Student and school characteristics were included in this analysis. 

• A full explanation of the research methodology used in this study is available at: 

www.edsource.org/pub_abs_simstu05.cfm 

Interpreting the findings 

• The study shows four domains that are most highly correlated with higher API scores: prioritizing 
student achievement, implementing a coherent, standards-based instructional program, using 
assessment data, and ensuring availability of instructional resources. That does not mean those 
domains have actually caused those higher scores. Rather, it indicates that schools that report 
more strongly that they have implemented more of the practices included in each of the four 
domains have, on average, higher API scores than schools that report fewer of the practices. 

• The school practices associated with these four domains tend to occur together (i.e., schools high 
on one domain tended to be high on others). 

• Practices reflected in the other three domains-while certainly important in building social capital 
and community at a school-did not show up as strongly in differentiating the lowest-performing 
schools from the highest in the study sample. 



Teachers have opened themselves up to learning from each other, sharing ideas, and 
expressing their need for more training to teach English learners. Special Education, 
and high-needs students. The communication is open, honest, and respectful. 

-A school principal 



Overview of the Research Process, 
on this page to understand how the 
analysis was done.) 

Within our sample of elementary 
schools, certain “effective schools” 
domains proved to be significant in 
distinguishing the responses of teach- 
ers and principals in the highest- 
performing schools from those in the 



lowest-performing. Those included: 

• Prioritizing student achievement; 

• Implementing a coherent, standards- 
based instructional program; 

• Using assessment data to improve 
student achievement and instruc- 
tion; and 

• Ensuring availability of instruc- 
tional resources. 



Prioritizing student achievement 

This domain examined the importance 
schools and districts place on setting 
clear, high, and measurable expecta- 
tions for student achievement. Several 
research studies over more than a 
decade have suggested that communi- 
cating such expectations has a positive 
effect. Some of that research has par- 
ticularly focused on high-performing, 
high-poverty schools. Common char- 
acteristics among those schools often 
include high expectations communi- 
cated in concrete ways and established 
systems to assess regularly the progress 
of individual students. 

This study asked both teachers and 
principals about the extent to which 
their school and district communi- 
cated high expectations and took 
responsibility for student achievement. 
It also asked about the degree of prior- 
ity given by teachers, the principal, and 
the district to meeting API and federal 
adequate yearly progress (AYP) targets 
for subgroups of students (such as by 
race/ ethnicity and income level). 

Survey results showed that the 
schools with teachers and principals 
who indicated higher expectations for 
students had, on average, higher API 
scores than similar schools with lower 
reported expectations. One aspect of 
this is a shared culture within the 
school regarding the value of improv- 
ing student achievement and a sense of 
shared responsibility for it. The 
responses in this category included 
teachers and principals reporting that 
their school has a vision focused on 
student-learning outcomes and that 
teachers take responsibility for and are 
committed to improving student 
achievement. 

Principal behaviors are also note- 
worthy. Teachers and principals in 
high-achieving schools were more 
likely to report that the principal 
communicates a clear vision for the 
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school, sets high standards for student 
learning, and makes expectations clear 
to teachers for meeting academic 
achievement goals. 

The practices in this domain go 
beyond a shared culture, however, to 
specific expectations. Both teachers 
and principals at higher-performing 
schools reported that their school has 
well-defined plans for instructional 
improvement and that they make 
meeting the states API goals and the 
No Child Left Behind AYP goals a 
priority. And they reported that their 
schools set measurable goals for 
exceeding API growth targets for 
student subgroups. These principals 
also indicated that their school’s 
statewide and similar schools rankings 
on the API influence schoolwide 
instructional priorities, and they 
reported that they are clear about their 
district’s expectations for meeting API 
and AYP growth and subgroup targets. 

Implementing a coherent, standards- 
based instructional program 

This domain looked at the extent to 
which a school’s curriculum and 
instruction are both coherent and 
aligned with state standards. The 
survey focused on the core areas of 
mathematics and English language 
arts. Teachers answered questions 
about the amount of time spent on 
each subject, the extent to which the 
two core subjects are protected from 
interruption, and whether math and 



language arts are integrated with other 
subjects. They also indicated which 
English and math curriculum pack- 
ages they used in their own classrooms 
and how frequently they used those 
packages. And teachers answered ques- 
tions about alignment and consistency 
in curriculum and instruction, 
planning, and materials. Principals 
answered questions about their prac- 
tices and their perceptions about the 
school district’s expectations related 
to curriculum coherence. 

The findings regarding the value 
of a coherent, schoolwide curriculum 
program are consistent with previous 
research. Studies of this issue going 
back two decades repeatedly point 
to curriculum coherence within a 
school — and among school staff — as 
being correlated with higher student 
performance. (For additional back- 
ground, see the To Learn More box on 
page 20.) 

In this study, teachers who work in 
schools that on average performed 
better were more likely to report 
schoolwide instructional consistency 
within grades and curricular align- 
ment from grade-to-grade. The kinds 
of practices teachers reported using 
included examining the scope and 
sequence of curriculum topics and 
reviewing a grade-level pacing calendar 
that sets out a timeline for instruction. 

Alignment with state academic 
standards also appears to be reported 
more often by teachers in schools that 



/ think the answer lies in the personnel: I mean the principal and the teachers doing 
what they need to do for children, using the data, implementing the programs. I use the 
word ‘relentless’ a lot. They’ve got to be relentless about not accepting anything but 
learning from the children. They’re not going to let the children fail; they’re going to 
make them learn. 




superintendents say: 



High expectations are part of a 
district cuiture. 



Of the 20 superintendents interviewed, three cited the 
estabiishment of ciear, high expectations as one of the 
most important factors in improving student achieve- 
ment over the iast four years, and 12 more at ieast 
mentioned this as a strategy their district had used. 
Severai superintendents spoke not oniy of the reia- 
tionship between student achievement and a schooi’s 
cuiture of high expectations, but aiso about the impor- 
tance of hoiding principais and teachers responsibie 
for meeting high standards. As one superintendent put 
it: “Whiie we were raising expectations for chiidren, we 
were raising expectations for the aduits.” 

Another superintendent described the chaiienges 
schooi staffs faced in this process and the extent to 
which the process changed some fundamentai 
assumptions on the part of aduits. “We had to reaiiy 
iearn to accept that we were maybe dumbing down. 
We were teaching to the iowest student, it was a 
tough beginning, but teachers after a whiie began to 
accept that and say, ‘OK, I can’t keep them here, i 
have to move on. Then what instructionaiiy do i need 
to do? What strategies work?”’ 

in one district, a key part of their strategic pianning 
process was gaining broad-based buy-in for reforms. 
The superintendent noted that the pian-which heid 
everyone responsibie for improving student achieve- 
ment-had “a ciear expectation that starts very much 
at the top.” 

Another district estabiished a “superintendent’s 
advisory councii,” made up of “key teachers and 
administrators from aii the schoois.” Monthiy meet- 
ings improved the district’s understanding of how 
strategies and poiicies were being impiemented 
inside ciassrooms as weii as what support and 
deveiopment teachers and principais needed. They 
aiso heiped the district buiid a sense of community 
in which everyone participated in the conversation 
regarding student achievement. 



-A superintendent 
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superintendents say: 



A districtwide curricuium adoption 
is important, but impiementation 
varies. 



Overall, 18 superintendents out of 20 mentioned the 
use of a curriculum package as a strategy they used 
to Improve student achievement. Five ranked this 
among their most Important strategies. 

Several district superintendents said that curriculum 
programs ensured Instructional consistency In their 
schools. As one noted: “In the past, every single 
school could have a different Instructional program 
for language arts. Some taught phonics, some didn’t; 
some used whole language, some didn’t.” 

Another administrator said that prior to the imple- 
mentation of its new curriculum program, the district 
was “less than organized or effective in providing a 
comprehensive, consistent reading program to our 
students. It was probably far too hit-and-miss, 
depending on teachers and schools.” 

The superintendents’ comments also conveyed a 
tension between strict adherence to a curriculum 
program and adapting instruction to meet student 
needs. Some emphasized their belief that uniform 
implementation was crucial. Others supported 
giving teachers some flexibility and described how 
that had occurred in a deliberate way. For some 
districts, pacing plans were a key strategy, giving 
the district an effective way to monitor and assess 
how teachers implement specific curriculum strate- 
gies to develop specific student skills. Here too, 
however, districts seemed to differ in their use and 
reliance on a pacing plan and on the extent to 
which they altered that plan based on school-level 
feedback. 



[W]e adopted one language arts program for K-8, and that was a major accomplishment... 
Plus, if the district has common expectations...uses a common textbook and common 
assessments, you can design professional development for everybody. 



are, on average, higher-performing. 
This includes classroom instruction 
being guided by state academic stan- 
dards and schools having identified 
essential standards. Teachers also report 
that the school’s curriculum materials 
in math and English are aligned with 
state standards and that they frequendy 
map state curriculum standards onto 
their classroom lesson plans. 

District actions, as reported by 
both teachers and principals, also 
appear to have some relationship to 
student performance. For example, 
teachers at higher-performing schools 
more often report that their district 
addresses the instructional needs 
of English learners at their school. 
Principals who reported a strong 
district role in this domain are also 
from higher-performing elementary 
schools. These principals said the 
district has a coherent, grade-by- 
grade curriculum for all schools and 
expects the principal to ensure imple- 
mentation of the curriculum. These 
principals report that the district has 
clear expectations for student 
performance aligned with the 
adopted curriculum and that it evalu- 
ates the principal based on the extent 
to which instruction in the school 
also aligns. 

Using assessment data to improve 
student achievement and instruction 

The use of data by teachers, princi- 
pals, and districts was perhaps the 
most intensively examined domain in 
our study, at least in terms of the 
number of survey questions. Under 
the general topic of data and assess- 



-A superintendent 

ment, questions addressed the types of 
assessment data teachers and princi- 
pals received, as well as how they used 
these data. We categorized the types of 
assessments as follows: 

• CST and CAT/6 assessment data, 
the state’s standardized tests admin- 
istered each spring. 

• CELDT (California English Lan- 
guage Development Test), an armual 
assessment of English learners. 

• Curriculum program assessments. 

• District-developed assessments. 

• Other commercial assessments. 

• Assessments created by individuals 
in a school. 

Teachers’ responses were organized 
based on I) the frequency with which 
they reviewed assessment data gener- 
ally, and 2) the extent to which they 
used the specific data types to monitor 
student performance and inform their 
instruction. 

The analysis of principals’ 
responses reflected different ques- 
tions, including their use of specific 
types of assessments and the extent to 
which they used each type to monitor 
achievement, address student progress, 
inform schoolwide instructional 
strategies, and monitor and evaluate 
teacher instructional practices. Prin- 
cipals were also asked about the 
influence of district expectations for 
improving student achievement, and 
about incentives and activities specifi- 
cally targeted at raising CST and 
CAT/ 6 scores. 

Both principals and teachers also 
responded to a set of questions about 
the extent to which they addressed 
student achievement by subgroup. 
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A substantial amount of research 
on the use of assessment data — and its 
relationship to student achievement — 
has been conducted since 2000. Much 
of this work has focused on imple- 
mentation issues related to high-stakes 
testing and on teachers’ use of data to 
inform their instruction. 

This study, by contrast, shed more 
light on how principals and school 
district officials are using student 
assessment data. One practice that 
showed a strong positive correlation 
with API scores among our sample of 
elementary schools was the extensive 
use of student assessment data by the 
district and the principal in an effort 
to improve instruction and student 
learning. As an example, principals 
from better-performing schools 
more often reported that they and 
the district use assessment data 
from multiple sources — curriculum 
program and other commercial assess- 
ments, district-developed assessments, 
and the CSTs and CAT/ 6 — to evalu- 
ate teachers’ practices and to identify 
teachers who need instructional 
improvement. They also reported 
frequently and personally using assess- 
ment data to address the academic 
needs of students in their schools, 
including using this data to develop 
strategies to help selected students 
reach goals and to follow up on their 
progress. In addition, they review this 
data frequently both independently 
and with individual teachers. 

These same principals reported a 
clear understanding of their district’s 
expectations for improving student 
achievement. They said that their 
districts expect that all schools will 
improve student achievement and 
evaluate principals based upon that. 
The principals reported that the 
district also provides support for site- 
level planning related to improving 
achievement. 



Findings on use of curricuium packages prompt extended anaiysis 



One of the strongest initial findings in this study was a correlation between a school’s API score and 
having a coherent curriculum and instructional program. Among the questions teachers were asked 
was one regarding which curriculum programs they use in their classroom and how often. When the 
data were analyzed, it appeared that the choice of curriculum program correlated with API. 

An extended regression analysis again held constant for student demographics and included a more 
specific definition of intensity of usage. That analysis found that for English language arts, using the 
Open Court curriculum program schoolwide appeared to distinguish between higher- and lower- 
performing schools.* The analysis found that while high-intensity use of Open Court mattered (i.e., all 
teachers in the school reported using Open Court daily), not all schools using Open Court intensively 
were higher-performing. Open Court appeared to be most effective when it was: 

• combined with a coherent, schoolwide, standards-based instructional program; and 

• combined with the frequent use of student assessment data to improve instruction. 

Open Court is one of two main English language arts curriculum packages currently approved by the 
State Board of Education. It was used as the primary English curriculum program by only one-fourth 
(72) of the schools in the study’s sample. But in 80% of those 72 schools, all teachers in the school 
reported using it daily. Most of the remaining schools in the sample said they used Houghton Mifflin 
for reading, with about two-thirds classified as high-intensity users but just one-third having all teach- 
ers reporting daily use. Some schools using Houghton Mifflin were also among the highest-performing. 

Teachers were also asked about the math curriculum that they used. In our sample, several different 
math curriculum programs appear to be in use, and relatively small percentages of teachers report 
using any one program daily. The analysis did not establish an association between the use of a 
specific math program and school API. 

"EdSource is not affiiiated in any way wittr any particuiar pubiistier. Neither EdSource nor this study in any way endorses one 
particuiar curricuium program over another, or one pubiisher over another. 



IVe moved to a standards-based curriculum with assessments tied to our report card. 
This led to pacing schedules, structured conversations about instructional strategies 
and student performances, and targeted achievement goals for the entire school, 
classrooms, and individual students. 

-A principal 



In schools where assessment data 
from the CSTs and CAT / 6 influence 
schoolwide attention to improving 
student achievement, the API score 
also tends to be higher. Teachers in 
these schools reported receiving CST 
and CAT / 6 test data in a variety of 
formats: for all students in their grade 



level; disaggregated by specific skills 
for all students in their classrooms; and 
disaggregated by student subgroup for 
students in their classrooms. Principals 
reported using the CST and CAT/ 6 
data for the following purposes: 

• To examine schoolwide instruc- 
tional issues; 
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superintendents say: 



Districts have deveioped expertise 
at using assessment data to 
improve student achievement 
and instruction. 



All but one of the 20 superintendents Interviewed 
mentioned “use of data and assessment” In explaining 
their strategies to Improve student achievement. Nine 
saw It as one of the most Important strategies they had 
Implemented over the last four years. Several said they 
had created data analysis positions and hired addi- 
tional personnel to help school staffs understand and 
apply assessment results. For the most part, however, 
these superintendent comments did not focus on the 
evaluative or summative assessment data provided 
through the state testing system, but rather on forma- 
tive assessments-tests used to evaluate ongoing 
student progress and Inform classroom Instruction. 

One superintendent described how the strong principals 
In the district worked with teachers to analyze the results 
of reading assessments given every six weeks. The prin- 
cipals sit down with grade-level teams of teachers to 
analyze how their students did collectively and Individu- 
ally. The goal Is not to accuse anyone but to look at what 
can be learned from students’ performance. The teach- 
ers whose students do well on one skill or another share 
the practices that seem to be successful. 

Another superintendent described a process, 
adopted from other research, that uses a “pyramid of 
Intervention. ...You ask three critical questions: What 
has the student learned? How do 1 know that the 
student has learned It? And what do I do If the 
student hasn’t learned It? That basically focuses our 
district’s whole Instructional program. What It means 
Is that you make data-driven Instructional decisions.” 

Many of these district officials placed particular em- 
phasis on the ways that principals, and by extension 
teachers, evaluated and worked with student subgroup 
data. Some said their district had provided specific 
professional development for principals toward that end. 






When we’ve gotten negative data, we have started being transparent about it and kind 
of ready pubiicizing it... foiks actuaiiy respect the fact that we’re out front and open. 

-A superintendent 

i’ve met individuaiiy and by grade ievei with teachers to review data, conduct 
professionai conversations based on student data, and set instructionai priorities. 



• To develop strategies for moving 
students from “below basic” and 
“basic” to “proficient”; 

• To compare grades within the 
school; 

• To identify struggling students and 
evaluate their progress; and 

• To inform and communicate with 
parents. 

(Note that test performance levels 
are labeled “far below basic, below 
basic, basic, proficient, and advanced,” 
with the statewide goal of all students 
achieving at least “proficient” in each 
subject tested.) 

Ensuring the availability of instructional 
resources 

School-level financial data are not 
available in California. As a result, the 
study could not look at financial 
records, such as expenditures, as part 
of its examination of schools’ instruc- 
tional resources. Instead, we defined 
“resources” broadly to include person- 
nel, their qualifications, and the 
availability of decent facilities and 
adequate textbooks. 

This research domain included 
survey data on the credentials and 
experience of educators plus teachers’ 
responses on the availability of class- 
room materials. We also considered 
principals’ perceptions of a number of 
different types of resources, including 
most notably the skills, knowledge, 
and attitudes of the teaching staff at 
the school. Principals were also asked 
about the extent to which the district 



-A principai 

provides support for facilities and 
instructional materials, any provision 
of longer school day or year, and the 
school’s access to qualified support 
personnel. 

Most education experts agree that 
teacher quality is central to student 
achievement. However, researchers 
exploring this question struggle with 
how to define teacher quality, often 
settling for those attributes most easily 
measured, namely teachers’ years of 
experience and credentials. Those 
aspects of teacher quality were among 
the factors examined in this study. The 
findings indicate that the schools 
where more teachers reported having 
regular or standard certificates for 
California also had, on average, higher 
API scores. Years of educator experi- 
ence also mattered. Teachers from 
schools with higher APIs were more 
likely to have at least five years of full- 
time teaching experience. Principals’ 
years of experience also correlated 
with higher school achievement. 

For another perspective on teacher 
quality, the survey asked principals to 
indicate the extent to which their 
school’s teaching staff possessed some 
less often measured and less easily quan- 
tified characteristics. API performance 
was on average higher in the schools 
where principals reported that a larger 
proportion of their teaching staff had 
the following qualities (the most signif- 
icant ones are listed first and so on): 

• demonstrated ability to raise stu- 
dent achievement. 
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• strong content knowledge, 

• good fit with the school culture, 

• training in curriculum programs, 

• ability to map curriculum standards 
to instruction, 

• supportive of colleagues’ learning 
and improvement, 

• able to use data from student 
assessments, 

• familiar with the school community, 

• excited about teaching, and 

• familiar with state standards. 

Principals responded to a variety of 
other questions related to the resources 
their district makes available to their 
school. Those more likely to be from 
high-performing schools more often 



lated with API scores, were not nearly 
as strong as the other four domains in 
differentiating lower- from higher- 
performing schools. Interestingly, 
these three share a common thread — 
they are all domains that tend to 
contribute to the social capital at a 
school, the positive interactions 
between students, teachers, parents, 
and principals that help a school create 
a sense of community. These things 
are clearly important to supporting a 
positive learning environment, but this 
study suggests that they are less 
powerful in explaining what the 
higher-performing schools do differ- 
ently from the lower-performing ones. 



Our district supported significant staff changes at our schooi through transfer, 
resignation, and retirement. 

-A principai 



said their districts ensure that the school 
has up-to-date instructional materials, 
support to provide supplementary 
instruction for struggling students, and 
enough instructional materials for all 
students. The same was true for teachers 
who tended to confirm these percep- 
tions, reporting that every student in 
their classroom has a copy of the 
current version of the textbook in 
language arts and math and that the 
principal ensures that the teachers have 
adequate classroom materials. The 
principals also reported a strong under- 
standing of what their district expected 
from them in terms of facilities upkeep 
and management and that their district 
provided adequate support in the area 
of facilities management. 

Three additional domains showed 
weaker correlations with API 

The study included three additional 
domains that, while positively corre- 



Enforcing high expectations for 
student behavior 

A substantial body of research since 
1990 has characterized a positive 
school climate as a basic building block 
for school effectiveness, particularly in 
high-performing, high-poverty schools. 
An important component of that is an 
ordered and disciplined environment. 
More recently some researchers have 
highlighted the vital role the school 
principal plays in this regard. 

In this research study, the examin- 
ation of order and discipline was 
limited to questions about the school’s 
establishment and enforcement of 
policies related to student behavior. 
Both principals and teachers reported 
on the extent to which the school 
created an orderly and positive en- 
vironment for student learning, 
including such areas as attendance 
policies, enforcement of rules, and 
respect for cultural differences. A 




superintendents say: 



Principal leadership and 
management are keys to building a 
strong teaching staff at a school. 

The superintendents interviewed spoke at iength 
about the relationship between an effective principal 
and the quality of teaching that occurs at a school. In 
particular, they connected the attributes of the 
school leader to the type of staff they are able to 
develop at a school. 

One official said the district looked for principals who 
“knew instruction and knew how to facilitate reform, 
but could do it while sustaining relationships with 
their staff.” As another noted, while “knowledge of 
instruction” is essential, principals also have to “be 
able to motivate staff. They have to be able to get 
staff on board. They have to be strong enough to 
monitor and make the tough calls and to correct 
where they need to.” 

Several superintendents remarked that their most 
effective principals frequently spent time in class- 
rooms working directly with teachers. To improve 
instruction, principals must spend “a great deal of 
time observing, monitoring, giving feedback so that 
they can really be responsive to the needs of our 
teachers,” according to one official. Yet effective 
principals do not need to be dictatorial. Instead they 
“evaluate their own practice and lead discussions 
about their practice in an evaluative way for continu- 
ous learning and continuous improvement rather 
than blame.” 
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IVe make it extremely difficult to get a job here. We say we’re going to see 1,000 
teachers to find 100, and we do. We’re going to he very, very picky because we know 
that’s what makes a difference. 



simple tabulation of the survey 
responses as a whole shows that the 
respondents, and particularly the prin- 
cipals, were predominantly positive on 
these questions. Depending on the 
question, between 70% and 90% of 
teachers and more than 90% of prin- 
cipals agreed that each was effectively 
in place at their schools. 

Because this study was looking for 
the practices that differentiate high- 
from low-performing schools, this 
relative unanimity could explain why 
this domain did not show up strongly 
in the analysis. The responses suggest 
that high expectations for student 
behavior are important at most of the 
schools in the study, regardless of their 
academic performance. 

Involving and supporting parents 
Parent involvement has also been seen 
by at least some researchers as a neces- 
sary but not sufficient component 
of effective schools. Measuring this 
particular aspect of schooling, how- 
ever, requires clearly defining which 
aspects of parent involvement are of 
interest. For example, volunteering at 
school and helping with homework 
are both aspects of parent involve- 
ment, but each is measured differently 
and has a different relationship to 
student achievement. Some recent re- 
search indicates that involving parents 
in student learning is particularly 
important for student achievement. 

For this study, the domain related 
to involving parents included subdo- 
mains that looked broadly at just two 
aspects of this complex subject: the 
school’s active engagement with 



-4 superintendent 

parents and its support of parents and 
families. The 29 questions included in 
the surveys did not ask about numbers 
of parents involved or the number of 
hours they volunteered. Instead, teach- 
ers were asked about: I) their practices 
involving parents in students’ educa- 
tion; 2) the district’s success in 
building the community’s confidence 
in the school; and 3) the principal’s 
relationship with parents. 

Principal questions were more 
comprehensive. They included the 
extent to which the school: I) involved 
parents in students’ education through 
mechanisms such as parent-teacher 
conferences, schoolwide events, and 
translators for non-English speaking 
parents; 2) worked to engage parents in 
schoolwide decisions and activities; and 
3) provided support services to parents 
and families, including English as a 
Second Language (ESL) classes, health 
services, and assistance programs. 

For the elementary schools in our 
sample, all of which serve low-income 
families, practices designed to involve 
the parents in their children’s educa- 
tion were more positively correlated 
with higher school performance than 
were other efforts to include parents. 
Examples of these practices were the 
frequency of special subject- area 
events (like math or English language 
arts workshops) held at a school and 
the frequency with which parents 
provided instructional support in 
classrooms. 

More generally, a positive correla- 
tion also emerged for schools with 
teachers that reported most strongly 
that their district builds the commu- 



nity’s confidence in the school and 
their principal builds strong relation- 
ships with parents. 

These findings may suggest that at 
schools serving low-income children 
and families, parent involvement 
strategies should be centered on the 
school’s instructional program and 
the child’s progress. This approach 
could be seen as naturally comple- 
mentary to the four practices that 
correlated most positively with school 
performance. All four have student 
learning and academic achievement as 
their focus. 

Encouraging teacher collaboration and 
providing professional development 
This domain examined three different 
areas related to the professional envi- 
ronment in the school and looked at a 
wide range of activities by teachers, 
principals, and districts. The three 
areas were: I) teacher collaboration 
and professionalism; 2) professional 
development; and 3) the hiring, evalu- 
ation, and firing of teachers. This area 
of questioning largely focused on how 
schools go about hiring and developing 
a capable teaching staff, while the 
resources domain noted above asked 
about the presence of capable teachers. 

Research into the ways that teach- 
ers interact and the effect of that 
interaction on student learning goes 
back to at least the early 1980s. The 
body of research is complex, raising 
questions about both positive and 
negative aspects of teacher collabora- 
tion and the challenges involved in 
developing a professional community 
in which that interaction leads to 
improved student performance. 

Within this study, the topics of 
teacher collaboration and profession- 
alism were addressed with survey 
questions that looked at the extent to 
which teachers felt they had influence 
over schoolwide decisions. It also 
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asked teachers and principals about 
opportunities for teacher collabora- 
tion on curriculum and instruction, 
including for English learner students; 
their perceptions that teacher pro- 
fessionalism was supported and 
encouraged within the school and by 
the district; and the extent to which 
they experienced a continuous learn- 
ing environment. 

Much of the research regarding 
professional development, on the 
other hand, has largely focused on 
which program characteristics are 
most important in providing high- 
quality professional development 
that will lead to increased student 
achievement. Consistency and a focus 
on subject-matter content are often 
cited as particularly important. Most 
of this research has been done in the 
past decade. 

The second strand in this domain 
related to the development of educator 
capacity through professional develop- 
ment. A particular focus of the survey 
was on the adequacy, influence, and 
value of a large number of different 
teacher professional development 
opportunities, including training linked 
to standards generally, specific curricu- 
lum programs, instructional strategies, 
the use of data to inform instruction, 
and noninstructional issues. Teachers 
were also asked about their participa- 
tion in coaching and modeling 
activities. Principals answered questions 
about the professional development 
opportunities their district provided to 
teachers and to them personally, and 
the value they ascribed to the latter. 

Finally, the study explored the 
principals’ perspectives on the hiring, 
evaluation, and firing of the teaching 
staff at their school. Principals were 
asked about their district’s success in 
building and maintaining a strong 
teaching staff and their own capacity 
to evaluate teacher performance. They 



also answered questions about their 
ability to hire and remove teachers, 
including district and school factors 
that influence that ability. 

Again, while this domain as a 
whole was positively correlated with 
API scores for the schools in the study, 
the relationship between professional 
development and API was relatively 
weak. This stands in contrast to the 
emphasis we heard in our interviews 
with the 20 superintendents. Many 
of them characterized professional 
development as a key strategy for 
instructional improvement. One 
possible explanation for this discon- 
nect may be that the survey questions 
about professional development 
included in this domain were more 
general in nature rather than being 
closely aligned with the domains we 
found to be significant, such as use of 
data and instructional program imple- 
mentation. Thus, they may not have 
captured the specific activities superin- 
tendents had in mind. 

Another view of the findings 
reveais strong verticai aiignment 
of expectations and accountabiiity 

Our research analysis examined the re- 
lationship of various “effective schools” 
practices to school API scores. Another 
way to look at the findings is to identify 
which actors at which levels of the 
system are involved in implementing 
each of these practices. While the focus 
of this study was at the school level, the 
results provided some evidence about 
the relationship between school district 
practices and school performance. The 
results also helped illuminate the 
changing role of the principal as the 
conduit between district and classroom 
and as a central player in the achieve- 
ment of that vertical alignment. 

It appears that in schools with 
higher APIs, both principals and 
teachers are more likely to report prac- 




superintendents say: 



Professional development is the key 
to changing classroom practice. 

The superintendents we interviewed expressed a 
strong beiief that their professionai deveiopment 
efforts were centrai to their abiiity to transiate the 
state’s achievement goais into district expectations, 
schooi impiementation, and uitimateiy to changes in 
ciassroom instruction. Overaii, seven of the 20 super- 
intendents cited professionai deveiopment as a key 
strategy, and another eight mentioned it as some- 
thing they had impiemented. 

Severai superintendents saw professionai deveiop- 
ment as significant for improving instruction and 
student achievement because it directiy affected 
ciassroom teaching. One said that extensive profes- 
sionai deveiopment gave teachers “tooi kits of 
strategies” that they “can puii out whenever they 
need it for whatever kind of iesson they’re using.” 

Severai districts used coaches to heip drive profes- 
sionai deveiopment into the ciassroom. One 
superintendent mentioned training coaches on 
specific content, ensuring that they were abie to 
“reaiiy get into the ciassrooms, work with grade-ievei 
teams, infiuence instruction, and assist teachers who 
aren’t abie to do the work.” This district aiso “puiied 
[aii coaches] out on a monthiy basis for a fuii day of 
training” to heip them meet teachers’ needs. Another 
superintendent said that after moving to a new 
curricuium, the district used coaches to heip teach- 
ers with various tasks, such as “pacing and buiiding 
iesson pians" and creating iessons “centered around 
the iearner.” 
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superintendents say: 



District practices and support can 
strengthen principai ieadership. 

The superintendents placed substantial emphasis on 
school principals and their role, with 18 out of the 20 
superintendents mentioning it. Three superinten- 
dents ranked principal leadership among the most 
important factors in improving student achievement 
over the last four years. They said that effective prin- 
cipals drove reform, held high expectations for 
student achievement, and provided instructional 
leadership and guidance. 

Their remarks also revealed the influence of district 
practices on the way principals do their jobs. For 
example, several superintendents spoke of efforts to 
transform the role and responsibilities of the princi- 
pal. One district had previously hired principals 
“based upon their operational knowledge and skills.” 
However, as demands of accountability required 
sharp improvements in student achievement, the 
district's notion of an effective principal evolved. 
“[The principal has to] know instruction, ... lead 
instruction, and ... make sure that instruction is good 
in classrooms." Another superintendent said princi- 
pals were expected to have a clear “vision for their 
school. They have to have an idea of where [they] 
want to go by the end of the year and howto bring the 
school along as a learning organization.”Yet another 
noted that in order for principals to be effective, 
“they’ve got to know that their supervisor supports 
them and is there backing them up.” 

District officials mentioned the value of principals 
acting as instructional leaders by knowing“whatgood 
instruction looks like and. ..what to expect." Another 
official mentioned the role of the principal in ensuring 
that a “school’s instructional program [is] focused on 
those standards and the articulation between the 
standards and what happens in each classroom.” 






tices that indicate: I) strong district 
leadership and support aimed at 
improved student achievement; 2) a 
redefinition of principal leadership 
targeted at the evaluation and 
improvement of school and student 
progress; and 3) teachers who take 
responsibility for student achievement 
based on state standards. Taken 
together, these examples suggest a 
strong vertical alignment between the 
actions of district leaders, school prin- 
cipals, and classroom teachers. All 
appear to share similar high expecta- 
tions for student performance and to 
collectively stress adult accountability 
for meeting those expectations. 

The superintendent interviews 
further reinforced these general 
impressions. They also made clear the 
extent to which the selected districts, 
all of which had high-performing 
schools, were actively engaged in 
trying to improve both classroom 
practice and principal leadership 
through an alignment of expectations 
and practice. 

While the study was not designed 
to directly examine the influence of 
district policies, principals at high- 
performing schools tended to perceive 
many aspects of their district’s role 
differently from principals at low- 
performing schools. Based on 
principals’ survey responses, it appears 
that districts may actively contribute 
to a higher API at these elementary 
schools in a variety of ways related to 
standards-based reform. Specifically, 
principals at high-performing schools 
said their districts were clear in their 
expectations that schools meet both 
the API and AYP growth targets for 
the school as a whole and for 
subgroups. They ensured that the 
school curricula in math and language 
arts were aligned with state stan- 
dards and that instruction was focused 
on student achievement. Principal 



responses also suggest that there may 
be a relationship between API per- 
formance and the extent to which the 
district provided schools with student 
achievement data and evaluated princi- 
pal performance and teacher practices 
based on that data. 

The Initial Findings also appear to 
reinforce a wealth of research that 
points to the school principal as the 
crucial actor in the effectiveness of a 
school. In general, researchers have 
found that schools with good leader- 
ship are more likely to have pervasive 
and sustained student learning. More 
specifically, research indicates that 
principals in effective schools use 
measurable goals to establish a culture 
of achievement, create a collective 
sense of responsibility for school 
improvement, and act as models of 
instructional leadership. In addition, 
they are persistent and innovative in 
obtaining resources to serve students. 

Among the schools included in this 
study, API scores were generally higher 
in schools with principals whose 
responses indicate that they act as 
managers of school improvement, 
driving the reform process and culti- 
vating a strong school vision. In 
particular, they were more likely to 
embrace the state’s academic standards 
and to ensure that classroom instruc- 
tion was based on them. They 
prioritized meeting and exceeding state 
and federal accountability targets for 
school performance. In addition, they 
reported personally and extensively 
using student data for instructional 
management purposes — not only to 
evaluate the progress of students, but 
also to examine schoolwide and teacher 
practices, develop strategies to help 
selected students reach goals, and iden- 
tify teachers who need instructional 
improvement. Finally, they ensured 
that teachers and students had 
adequate texts and classroom materials. 
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There isn’t an elementary principal in this district who doesn’t know what’s expected 
of them, what their work plan needs to have, ... and what’s going to happen when my 
deputy superintendent and! do tours of that campus. They know what we’re looking for. 

-A superintendent 



Teachers in schools with higher 
APIs indicate that they are more likely to 
take responsibility for student achieve- 
ment, again as achievement is defined by 
the states academic standards. They 
report using the standards for guiding 
instruction and ensuring curriculum and 
instruction alignment within and across 
grades. They also say that they are well 
trained in standards-based instruction. 
These teachers are also more likely to 
receive data from the states annual 
assessments (CSTs and CAT / 6). 

This study offers implications 
for policy and practice, including 
“staying the course” with 
standards-based reform 

In the course of this study, EdSource 
and its research partners from Stan- 
ford University, UC— Berkeley, and 
American Institutes for Research gath- 
ered extensive data on the practices 
and policies of elementary schools in 
California that serve high numbers of 
low-income and English learner 
students. The goal of this effort was to 
identify the practices that set the 
highest-performing schools apart 
from the lowest-performing ones. 
Since the release of the Initial Findings 
in October 2005 and the further 
analysis on curriculum package choice 
in April 2006, many people have 
asked us to articulate more specifically 
the practices reported by these schools 
and the implications of our findings. 

The first consideration in doing so 
is the larger policy context in which the 
elementary schools in our sample have 
been operating. The years since Cali- 
fornia adopted K— 12 academic 



content standards for its public schools 
in the late 1990s have been tumul- 
tuous and challenging for education 
leaders in schools, school districts, and 
state-level positions. Part of a national 
movement in school reform, the adop- 
tion of standards was a watershed 
event for California that paved the way 
for a new state assessment system and 
new methods of holding schools 
accountable based on those tests. 
These changes have taken time and 
have represented difficult work intel- 
lectually as well as politically. 

The findings from this study indi- 
cate that many of the policies and 
practices that have accompanied 
standards-based reform in California 
are correlated with improvement in 
achievement at schools with high 
proportions of low-income students. 
We believe that the overarching 
message from the findings is that state 
policymakers and local educators need 
to stay the course in terms of explicit 
expectations for student achievement 
and a process of public accountability 
that keeps those expectations at the 
forefront of schools’ efforts. 

Aligning school and district instruction 
with the state’s high expectations is a 
multifaceted challenge 

For educators and local communities, 
a central message of the study is that 
school practices and policies can make 
a difference in the achievement of low- 
income students. Some schools clearly 
do a much better job of enabling their 
students to learn the state’s grade-by- 
grade academic content standards 
than do other schools, regardless of 
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superintendents say: 



Districts use professionai 
deveiopment to aiign instruction 
with expectations. 



In discussing professional development, the super- 
intendents also reinforced the idea of vertical 
alignment. These superintendents characterized 
effective professional development as a strategy that 
starts vi/ith the district but has a tangible impact all 
the way down to the classroom. One mentioned the 
importance of professional development that was 
“extremely focused and sustained” at the “district 
level, at the principal level, at the teacher level, and 
at the paraprofessional levels.” 

Another noted that a district needs to ensure that 
teachers are using programs “the way they were 
designed to be used; and if not, that's a good place 
to start. Make sure that you go back and retrain your 
teachers. Oftentimes you adopt a new reading text, 
you do the publisher’s training, give them the book, 
and assume everybody’s using it the way it’s sup- 
posed to be. We found that is not the case.” 

The superintendents also acknowledged the 
complexity of providing professional development 
that teachers were able to incorporate into their 
daily practice. Several said they adopted a “chain-of- 
command” approach with coaches, principals, and 
teachers. One district first provided professional 
development to principals and coaches, who were 
then responsible for“training the teachers in a mean- 
ingful way.” The district also trained district office 
administrators, ensuring that they were able to 
support and monitor the coaches’ work at the school 
to ensure implementation. 
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In schools with higher API scores, it is more iikeiy: 



All express a commitment to student achievement on the state’s standards, coupled with specific 
practices to support that goal. 

• School sets measurable goals for exceeding the mandated API student subgroup growth targets. 

• School uses statewide and similar schools API rankings to Influence Instructional Improvement. 

• Teachers ensure that curriculum and Instructional materials are aligned closely with state 
academic standards by frequently mapping those standards onto weekly lesson plans. 

• Teachers ensure Instructional consistency within grades by using a grade-level pacing calendar. 

• Teachers ensure curricular alignment from grade to grade by examining the scope and sequence of 
curriculum topics. 

• School uses curriculum programs that, among other qualities, support ease of teacher Implemen- 
tation of curriculum coherence. 

• Principals and teachers have access to CST data In a variety of formats: for all students In their 
grade level; disaggregated by specific skills for all students In their classrooms; and disaggregated 
by student subgroup for students In their classrooms. 

• Principals frequently review, discuss, and use student assessment data from multiple sources- 
annual CSTs, regular curriculum program diagnostic tests, teacher- and district-developed tests to: 

• address with teachers the academic needs of students; 

• develop strategies to help selected students reach goals (moving them from below basic to basic 
to proficient) and to follow up on their progress; 

• examine schoolwide Instructional Issues and compare grades within a school; 

• evaluate teacher practices and Identify teachers needing Instructional Improvement. 

• School ensures that every student has a copy of the current English language arts and math 
textbooks. 

Principals report that their districts have the same intense focus, saying the district; 

• Has a coherent grade-by-grade curriculum that it uses for all schools, and the district expects the 
principal to ensure implementation of the curriculum. 

• Has clear expectations for student performance aligned with the district’s adopted curriculum and 
evaluates the principal based on the extent to which instruction in the school aligns with the 
curriculum. 

• Makes it clear it expects that all schools will improve student achievement and that schools will 
meet their API and AYP growth and subgroup targets. 

• Provides schools with student achievement data. 

• Evaluates the principal based upon improved student achievement and uses student achievement 
data to evaluate teachers’ practices. 

• Provides support for site-level planning related to improving achievement. 

• Ensures that the school has up-to-date instructional materials, support to provide supplementary 
instruction for struggling students, enough instructional materials for all students, and support for 
facilities management. 



student demographics. The findings 
showed some specific things schools 
do that can help explain the 250-point 
difference in API among schools serv- 
ing very similar groups of students. 

A shared culture that makes student 
achievement a top priority can 
improve performance 
The need to focus on student achieve- 
ment may seem self-evident, but K— 12 
educators, community opinion leaders, 
and parents often struggle to balance 
high expectations for academic achieve- 
ment in the core subjects of math and 
English language arts with other goals 
of schooling, such as socialization, arts 
instruction, exploratory learning, civic 
responsibility, and student health 
issues. This tension, which can be 
found at all grade levels, can be par- 
ticularly poignant when educators 
confront the many student needs at the 
elementary level. Educators know, 
however, that learning the core subjects 
well is critical for the future school 
success of their students, particularly 
those who face challenges due to family 
circumstances. 

This study found that staffs at the 
higher- API schools in our sample were 
more likely to report a variety of prac- 
tices aimed at making student 
achievement a shared priority and 
responsibility. These schools had a 
shared culture of student achievement 
that seems to permeate the work of 
teachers and principals and is strongly 
supported — if not demanded — by 
school district officials. The specific 
practices they reported are reflected in 
the box to the left. Through these 
practices, the schools expressed their 
commitment to measuring and im- 
proving student achievement based on 
the state’s academic standards and 
actively using curricula and instruc- 
tional processes toward that goal. 
Further, their school districts appear 
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to be taking actions that reinforce this 
commitment and support the capacity 
of both principals and teachers to 
meet their goals. 

The skills and capacity of educators 
make an important difference 
Raising student achievement in Cali- 
fornia at the same time that the 
student population is becoming 
increasingly challenged by language 
and poverty barriers requires a great 
deal of focused, hard work. The main 
findings of this study make this clear. 

Interviews with 20 superintendents 
further suggest that the solution, in their 
view, may also require actions beyond 
the kinds of support and accountability 
outlined by principals in their survey 
responses. These superintendents said 
that improved student learning also calls 
for a hard-nosed focus on hiring and 
training the best possible principals and 
teachers. In other words, people make 
the difference, and those people need to 
have the knowledge and capacity to do 
this challenging work. This dovetails 
with the study finding from survey 
responses that principals are more likely 
to be in schools with higher APIs if 
they reported that their teaching staff: 

• Has a demonstrated ability to raise 
student achievement; 

• Is knowledgeable about the state’s 
academic standards; 

• Has strong content knowledge; 

• Has been trained in the curriculum 
and instructional program; 

• Knows how to map standards to 
instruction; and 

• Is able to use data from student 
assessments. 

None of these skills comes auto- 
matically with a teaching credential. It 
is likely that strong professional devel- 
opment programs can help teachers 
develop these kinds of skills and 
content knowledge. Another implica- 
tion relates to distribution of teachers 



School districts and state policymakers may want to evaluate the 
availability to teachers of assessment data by student subgroup 



In the survey, both principals and teachers were asked if they received student assessment data disag- 
gregated by student subgroup. One finding was that teachers in schoois with higher API scores were 
more iikeiy to report that they received CST and CAT/6 data disaggregated by student subgroup. 

A simple tabulation of the principal and teacher survey responses to questions about subgroup data 
sheds some additional light on the availability of this data, even though it cannot be assumed that the 
variation in responses correlates with API performance. 

That tabulation shows that among the 257 school principals who responded, the vast majority (91%) 
said they received CAT-6/CST data disaggregated by subgroup. By contrast however, only 35% of all 
teachers in the sample reported receiving the same data for their classrooms. 

In light of this combination of information, school district and site leaders might find that making sure 
all teachers have access to student subgroup data would be worthwhile.The state ensures that districts 
receive subgroup data at the school level, but districts have to invest time and effort to disaggregate 
the data down to the classroom level. 



We’ve also given [principals] really specific training on how to have...courageous 
conversations with staff memhers, and how to evaluate staff memhers...and then even 
how to manage their own ego. 

-A superintendent 



among schools within a district. All 
the schools in our sample served high 
percentages of low-income students, 
but the schools with higher APIs were 
more likely to have strong teaching 
staffs, according to the school princi- 
pals. District programs and policies 
that deliberately and strategically 
build strong teaching staffs at all 
schools, but particularly at those serv- 
ing low-income students, will make 
an important contribution to those 
schools’ capacity to improve student 
achievement. Equally important is a 
district’s selection of school principals 
who come to the job well-qualified 
and who receive appropriate support 
and professional development. 

Increasingly, researchers and policy- 
makers are recognizing the important 



role school districts play in providing the 
leadership and accountability needed for 
teachers and principals to realize their full 
professional potential. To work effec- 
tively, districts in turn need state policies 
that support their leadership while hold- 
ing them accountable for improvement in 
their schools. They also need rational 
state accountability policies that are easy 
to understand and explain, plus curricu- 
lum materials that help them align all 
the necessary components and practices 
of school reform. Finally, they need 
resources, policy, and technical support 
to ensure that they can retain a high level 
of dedicated professional expertise at the 
district level. It takes qualified district 
personnel to effectively focus funding 
and resources on student achievement, to 
identify and implement high-quality 
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Survey results document professional development needs of principals and teachers at 
schools serving low-income students 



4s part of the Similar Students, Different Results study conducted byEdSource, Stanford University, UC-Berkeley, and AIR, both principais and teachers were asked 
about their professional development needs. Respondents were §ven a list of 10 possible professional development topics and asked to select their top three 
priorities. We cate^rized their responses based on their school's 2005 Growth API to determine the extent to which those needs might vary between high- and 
low-performing schools. * We also looked at the responses from schools currently participating in Program Improvement, an intervention program under NCLB. 



Professional development needs of principals vary depending on 

Regardless of school performance, principals’ top choice for additional profes- 
sional development was “using assessment data.” Given this unanimity, it is 
interesting to note that the second and third most popular choices diverged 
based on whetherthe principal was from a high-performing, a low-performing, 
or a Program Improvement school. Among principals in high-performing 
schools, the next two priorities were “evaluating teachers’ instruction” and 
“addressing multicultural/diversity issues,” both of which ranked near the 
middle for the principals in the other two groups. By contrast, the principals in 
low-performing schools collectively chose “training and instructional strate- 
gies for English learner (EL) students” and “developing a school plan or shared 
vision” as their second and third priorities. While the former was popular 
among high-performing principals as well, the latter showed up 10th. For 
Program Improvement school principals, the EL strategy was second; but the 
third priority was “implementing a standards-based curriculum.” 



their school’s performance 

Since it began standards-based reform efforts, California has invested in one 
major program for principal professional development. Assembly Bill (AB) 75 
training. This program requires participants to attend 160 hours of training in 
order to qualify for state funding, and the content must address three specific 
areas: leadership and support of student instructional programs; leadership 
and management for instructional improvement; and instructional technol- 
ogy to improve pupil performance. Absent a renewal or extension, this 
program ends in July 2006. It is unclear the extent to which the Legislature 
will continue to invest in principal professional development; but if it chooses 
to do so, this study’s findings indicate that a more flexible approach to 
professional development for principals that acknowledges their diverse 
capabilities and needs might be appropriate. Of course that assumes that 
the principals are accurate regarding the professional development that 
would be most helpful. 



Teachers report similar professional development needs regardless of school performance 



Professional Development Topic 


Priority Among Teachers 
in High-performing Schools 


Priority Among Teachers 
in Low-performing Schools 


Priority Among Teachers 
in PI Schools 


Instructional strategies for multiple learning styles 


1 


4 


3 


Language arts curriculum program 


2 


1 


1 


Mathematics curriculum program 


3 


3 


2 


Instructional strategies for English learners 


4 


2 


4 


English Language Development curriculum program 


5 


5 


6 


Curriculum standards 


6 


6 


5 



In contrast to principals, teachers from high- and low-performing schools 
and Program Improvement (PI) schools were relatively uniform in their 
top choices of professional development priorities, though they differed 
somewhat in the order of these choices, as noted in this table. 

The professional development topics least likely to be chosen as a top 
priority by teachers were classroom management and student discipline; 
understanding and using data from assessments to inform instruction; 
multicultural or diversity issues; and school improvement planning. 

One interpretation of this, consistent with the other survey responses, is 
that teachers are naturally focused on one thing-teaching. New and 
even more seasoned teachers have faced a considerable amount of 
work mastering the state’s new academic standards and working 
together to align curriculum. These requests for professional develop- 



ment cover the main content areas of English, math, and English 
Language Development-as well as the instructional strategies needed 
to more effectively teach all students and those for whom English is a 
second language. 

Our survey results also indicate that while teachers in higher-performing 
schools report receiving student assessment data and reviewing it with 
their principal, their main job in the school improvement process is the 
implementation of the standards-based curriculum in a demographically 
diverse classroom of students. The fact that teachers from both the 
higher- and the lower-performing schools in our sample requested more 
training on curriculum subjects suggests that both state-supported 
professional development programs and teacher preparation programs 
at universities would do well to maintain a strong emphasis on them. 



•This look at professional development requests was done separately from the main analysis of survey data. For this anaiysis only, we designated high-performing schoois as those with a 2005 
Growth API that was one standard deviation above the mean for the sample schools and low-performing schools as those with a Growth API one standard deviation below. 
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teacher and principal professional 
development programs, to provide 
instructional support for struggling 
students, and to analyze student assess- 
ment data to improve teacher and school 
performance. 

The state’s choice of curriculum 
programs can affect school performance 

In the decade since state leaders in 
California began taking a standards- 
based approach to the oversight of 
public education, they have increas- 
ingly aligned the states curriculum 
adoption process with that approach. 
The state’s academic content stan- 
dards have formed the basis for 
curriculum frameworks and textbook 
content. Beyond that, however, the 
State Board of Education (SBE) has 
increasingly used its textbook evalua- 
tion criteria to encourage textbook 
publishers who want to sell books in 
California to closely align their 
materials with the state’s vision for 
classroom instruction. 

In 2002 the English language arts 
adoption required publishers to not 
only provide textbooks aligned with the 
standards, but to also augment the texts 
with robust materials that provided 
extra guidance for educators, including: 
• A clear scope and sequence for 
organizing the instructional 
program; 



• A variety of assessment instruments 
for use by classroom teachers; 

• Suggestions for modifying the 
materials to ensure universal access 
for students with special needs, 
including English learners; and 

• Instructional planning and support 
for teachers that helps them align 
instruction with standards. 

Only two publishers — SRA/ 

Open Court and Houghton Mifflin — 
met the state’s expectations at that time. 

The SBE recently approved the 
evaluation criteria for the 2008 
English language arts adoption. Text- 
book publishers will now have two 
years to develop materials that fit these 
criteria. Given California’s importance 
to publishers, most observers expect to 
see many more programs available that 
meet the state’s criteria this time 
around. Once the state adopts text- 
books it finds acceptable, districts will 
then review and choose among the 
approved list based on those they 
believe best meet their needs in terms 
of emphasis and approach. In many 
districts that means a process, often 
involving teachers, that leads to a 
districtwide adoption, with a plan for 
implementation and appropriate 
professional development to make 
sure that school staffs know how to 
maximize the effectiveness of the 
materials within the classroom. 



Superintendents depend upon their schooi principais to act as instructionai ieaders who 
are ahie to heip support and evaiuate teachers. 

-A superintendent 

increasing teacher coiiahoration opportunities has improved the ... deiivery of instruction. 
Supporting the coiiahoration with professionai deveiopment reinforces the abiiity of 
everyone to improve. 

-A principai 



What sets this study apart 



This study is different because of the foiiowing: 

• it focuses on Caiifornia eiementary schoois 
serving high percentages of iow-income 
students. 

• The study sampie inciudes schoois across a 
wide range of performance ieveis, sizes, and 
iocations. 

• The teacher and principai participation rates 
are very high. 

• Survey responses are within the context of 
state accountabiiity and standards-based 
schooi reform poiicies. 

• Survey questions address actionabie ciass- 
room, schooi, and district poiicies and 
practices. 



The state goes about this process on 
a set schedule, with different subject- 
matter areas considered each year. For 
elementary students, however, there is 
likely none more crucial than the English 
language arts adoption now underway. 

In two years — once publishers 
submit their materials for considera- 
tion — many opportunities exist for 
public review and input as part of the 
state’s adoption process. In preparation 
for that, state policymakers and local 
educators might be well served to learn 
more about the reading materials 
currently in use in California and the 
conditions that appear to make specific 
curriculum programs more or less 
effective in the diverse schools serving 
low-income children. One question 
that deserves particular attention is 
how curriculum and instruction are 
being most effectively adapted to the 
needs of English learners. That inquiry 
can help shape the broad decisions the 
SBE will ultimately make regarding 
how reading and the rest of language 
arts will be taught in California’s 
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• I To Learn More 

On the EdSource website at 

www.edsource.org/pub_abs_simstu05.cfm, you’ll find 
a wealth of information about the Similar Students, 
Different Results study, including: 

• The Initial Findings, released in October 2005. 

• Additional findings released in April 2006 related to 
curriculum adoptions. 

• Technical appendices that describe the research 
methodology and provide more detailed findings. 

• Copies of the actual surveys used with both teachers 
and principals. 

• Documents that address frequently asked questions 
(FAQs) about the findings generally and the issue of 
parent involvement in particular. 

• A comprehensive bibliography that includes the 
“effective schools” research used to develop the sur- 
veys and additional research that informed this study. 

• Biographical information about the research team. 

• Press coverage the study has received since its 
release. 

Copies of this study can be downloaded from the 
EdSource website. 

elementary schools, as well as the subse- 
quent district choices about their 
curriculum adoptions. 

The findings from this study suggest 
that this curriculum adoption process 
and the actual materials selected can have 
a real effect on schools’ ability to improve 
student achievement. State and district 
decisions should, to the extent possible, 
be based on evidence about what works 
to improve student achievement. 

School practices can make a 
difference in the achievement 
of low-income students 

Across California, schools serving similar 
types of student populations can vary 
widely in how well they score on the API. 
The 257 elementary schools (serving 



135,673 K— 5 students) included in the 
Similar Students, Different Results study were 
drawn from a fairly narrow band in terms 
of student demographics (percent low- 
income, English learner, ethnic/ racial 
subgroups). Yet their 2005 Growth API 
scores varied by about 250 points. 

A school’s API score reflects how well 
its students are performing on the annual 
California Standards Tests. Such tests are 
certainly not the only way for a school to 
measure how well its students are master- 
ing the rigorous academic content of the 
state’s grade-by-grade standards; and they 
also do not measure the other important 
things that elementary students may be 
learning at school — about art and science 
and music, about citizenship and toler- 
ance of differences, about themselves and 
their sense of place in the world. 

But an elementary school’s API score 
provides the state and the public with a 
consistent and easy way to grasp infor- 
mation about the progress its students 
are making toward mastering the impor- 
tant math and reading and writing skills 
that will enable them to succeed academ- 
ically in later grades. For this reason, 
among others related to accountability, a 
school’s API score represents an impor- 
tant measure of student learning. 

The range of API scores in our 
sample suggests that while the socioeco- 
nomic backgrounds of students is one 
predictor of academic achievement, it is 
not the sole predictor. What schools 
do — and what resources they have to do 
it with — can make a difference. With 
that in mind, the interrelated practices 
identified in this study may help schools 
in their efforts to improve student 
achievement. Further, the insights from 
the superintendent interviews may 
provide district leaders with some 
concrete examples of strategies to 
consider as they steer and support their 
schools’ improvement efforts. QD 




Trish Williams 

EdSource Executive Director 

2005-06 EdSource Board of Directors 

Ray Bacchetti, President 

Scholar in Residence, Carnegie Foundation 
for the Advancement of Teaching 

Davis Campbell, Vice President 

President, California School Boards Association 
Governance Institute 

Lawrence 0. Picus, Fiscal Officer 

Professor, Rossier School of Education, University 
of Southern California 

M. Magdalena Carrillo Mejia, Secretary 

Superintendent, Sacramento City 
Unified School District 

Pam Brady 

President-elect, California State PTA 

Kathleen Cha 

President, American Association 
of University Women-California 

Christopher Cross 

Chairman, Cross & Joftus, LLC 

Gerald C. Hayward 

Partner, Management Analysis & Planning, Inc. 

Santiago Jackson 

Assistant Superintendent, Division of Adult 
and Career Education, Los Angeles Unified 
School District 

Jacqueline Jacobberger 

President, League of Women Voters of California 

Martha Kanter 

Chancellor, Foothill-De Anza Community 
College District 

Kelvin K. Lee 

Superintendent, Dry Creek Joint Elementary School District 

Paul J. Markowitz 

Teacher, Las Virgenes Unified School District 

John B. Mockler 

President, John Mockler & Associates 

Ted Olsson 

Retired Manager, Corporate Community 
Relations, IBM Corporation 

Amado M. Padilla 

Professor of Education, Stanford University 

Peter Schrag 

Contributing Editor, Sacramento Bee 

Don Shalvey 

CEO and Co-founder, Aspire Public Schools 

Krys Wulff 

Director-at-large, American Association 
of University Women 



650/917-9482 



o 

520 San Antonio Road, Suite 200, Mountain View, CA 94040-1217 ■ 650/917-9481 ■ Fax: 
edsource@edsource.org ■ www.edsource.org ■ www.ed-data.kl2.ca.us 



© Copyright 2006 by EdSource, Inc. 



